Skip to content

fix(benchmark): bound and diagnose native receiver startup - #8214

Closed
loopx-agent wants to merge 2 commits into
codex/edgebench-integration-r29-20261107from
codex/native-receiver-startup-recovery-20261107
Closed

loopx-agent wants to merge 2 commits into
codex/edgebench-integration-r29-20261107from
codex/native-receiver-startup-recovery-20261107

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Native receiver preflight used to erase every failure into native_receiver_unavailable. A disconnected observer therefore aborted startup without identifying whether initialization, account read, or rate-limit read failed. Its initialization timeout also started after process creation, allowing slow spawning to escape the intended total observation window.

This change retains bounded local stage, error-category, attempt and elapsed-time diagnostics. It retries only the shared transport's exact disconnect/timeout codes, using at most three fresh disposable observers within the original 25-second IO window. Rejected RPCs, invalid accounts, local setup failures and unknown errors remain fail-closed. A remote error message that impersonates a timeout code cannot trigger retries.

The existing native reader accepts an optional monotonic deadline and checks it before spawning, after spawning, and before its initialized notification. Existing checkpoint readers that omit the option retain their relative-timeout behavior. Account reads still use refreshToken=false; no model request, credential refresh, session reset or solver-budget extension is added. Process cleanup may finish after the IO deadline.

Example synthetic failure output retains ok=false, code=native_receiver_unavailable and inference_invoked=false, with an attempt such as stage=account_read, error_kind=protocol_rejected. Raw exception text, native stderr, account identities and RPC bodies are never projected.

Validation on the final head:

  • Linux with the real SForge SDK: 22 tests pass, including all three generated startup commands, actual synthetic stdio subprocesses, successful retry after disconnect, three-attempt cap, forged remote timeout rejection, unchanged source profile and absent phase/scheduler writes on failure.
  • macOS: 19 readiness tests pass; three SDK-dependent checks skip because that SDK is absent there. The same three checks pass on Linux. Two separate optional real-native-binary checkpoint checks remain skipped; their explicit binary was not configured.
  • Existing checkpoint pagination, missing/foreign history, cursor and sealed-prefix compatibility checks: 26 pass.
  • Spawn-expiry and initialization-expiry counterexamples now prevent later native IO; cleanup/backoff and home preparation consume the same absolute window. Independent review is tracked separately.
  • Exact-head quick premerge: every direct and selected validation check passes (zero failures). Its overall result is manual_review_required solely because benchmark-sensitive changes require maintainer review; no merge is performed. An initial attempt lacked the TypeScript development dependency; the environment was installed without changing source or thresholds.
  • Actual SForge SDK installation into a new network-disconnected container: the final native-Goal frozen source imports without site packages, and the generated startup command correctly rejects initialization/account/rate-limit failures and passes a successful read. The success case uses a pre-expired execution deadline so no scheduler or model starts. Source and trial profiles are byte-identical before/after, disposable homes are removed, and model calls are zero. This uses a synthetic native stdio process, not a real account or a model service.

This is a bounded receiver-transport fix in the existing owner, with a small compatible deadline extension to its shared reader. The diagnostic categories remain local to this CLI; they do not create Goal states or a recovery/admission owner. This PR targets the integration branch that already contains receiver preflight. It changes no UI, scoring, evaluator feedback, active run or global account selection. A failed pre-execution trial without a session/checkpoint still requires an explicitly authorized cold replacement; this patch does not make it a warm recovery. No model adoption, benchmark-score improvement, or recovery of the original unclassified failure is claimed.

Main delivery

The receiver startup change and tests are carried by the main-target proposal #8262. This integration PR preserves its original review and source history.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…deadline

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6-astra · OpenAI · runtime_reported · reasoning_effort=ultra

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

使用 SForge 原生登录启动 LoopX worker 的操作者,会在模型开始前遇到接收端连接短暂断开。旧入口把所有失败压成同一错误并立即退出,操作者无法区分断连、账户拒绝和本地配置问题;新入口在原观察窗口内有限重试可恢复断连,并保留安全的阶段诊断。

独立实际子进程验证了短暂断连后的正常就绪,以及不可重试错误的立即拒绝;没有开始模型、改写原登录目录或启动求解计时。本 PR 不恢复已失败实验,不延长求解预算,不选择账户,也不证明模型效果或分数提升。原历史失败缺少阶段信息,无法倒推其具体原因;真实失败任务的显式替换与状态资格仍由原实验恢复流程处理。

改动思路 / Approach

沿现有接收端观察器收敛重试与脱敏,绝对期限由已有共享 stdio reader 执行,避免在 runner 另造通信或恢复状态机。当前切片完整覆盖启动前观察、有限重试、安全诊断及其真实生成命令;正式实验恢复和运行中 Turn 重试保持原 owner。

执行入口仍由 SForge 生成:先运行登录通路检查,成功后才进入原有阶段时钟。原账户与配置只复制到本次临时目录;每次重试创建全新观察进程,最多三次,共用同一个 25 秒 IO 窗口。无效账户、远端 RPC 拒绝、未知错误和本地配置错误立即结束;只有现有传输层精确产生的断连、超时代码允许观察器重试。原有求解器、评分服务与 Goal 调度不参与这次重试。

单纯增加日志能定位下一次失败,却不能恢复短暂断连;放大超时或无限重试则会隐藏成本。当前改动选择既有 owner 内的有限恢复,并把慢启动的期限检查下沉到已经负责启动进程的 reader,范围合理。

具体改动 / Changes

审查目标为 1a309ccb3278b68f79d674b66833fadffd8ad476,比较基线为 df594c8c3ba33f9380d261b74df6e1222bb97b28,目标分支是已有接收端入口的集成分支。规范依据:benchmark/runtime/RUNTIME.md,固定版本 df594c8c3ba33f9380d261b74df6e1222bb97b28。该版本的三个相关要求映射如下:startup-before-clock 已实现,由真实 SDK 生成命令验证失败时尚未创建阶段时钟;account-only-observation 已实现,实际子进程方法轨迹只有初始化、账户与额度读取,refreshToken=false;preserve-execution-authority 已实现,原目录、待处理状态和求解预算未被改写,正式恢复仍由原入口处理。

关键代码讲解

  1. _failure 对共享传输层产生的机器错误码作精确集合匹配。远端 RPC 的错误文字不会直接落入该集合;我让远端返回看似超时的字符串,实际仍只执行一次并拒绝。
  2. _observe_once 负责一次独立观察:复制所选配置、标记当前阶段、依次读账户和额度、清理临时目录。输出只有本地阶段、错误分类和耗时,没有账户响应、原始异常或 stderr。
  3. check_native_receiver 把准备、读取、重试退避纳入原窗口,且最多三次。断连后下一次成功可正常退出;持续断连最多三次后失败,不会建立新求解预算。
  4. native_checkpoint_reader 接受可选绝对期限,在启动前、启动返回后以及初始化完成后检查。未传该参数的 checkpoint 调用仍保留相对超时语义,避免顺带改变其他读取器。

对主干的风险 / Risks

独立审查在初版 80963767f4eae66bed381c7a14288bd8617c2103 重现了一个真实缺陷:进程启动耗时超过原观察窗口后,仍会使用旧的剩余时间发送初始化。最终头已在共享 reader 修复;相同反例现在不再发送初始化,初始化响应晚到也不会继续发送 initialized 通知。这一历史失败保留,不能被后来的通过记录抹去。

验证 结果及边界
独立基线/最终头实际 CLI 子进程对照,各 7 例 PASS:正常、短暂/持续断连、错误伪装、坏帧、账户不匹配;仅预期重试与安全诊断变化
独立共享 reader 3 例 PASS:慢启动到期、初始化到期、未传 deadline 的原行为
作者 Linux 真实 SDK 记录,已检查 PASS:22 例,包含三条生成启动命令;另 2 项可选真实 native binary 测试未配置而跳过
默认 checkpoint 兼容记录,已检查 PASS:26 例,分页、缺失/跨会话历史、游标和封存前缀
新断网容器的真实 SDK 安装与冻结启动命令,已检查 PASS:4 场景,原配置字节不变、临时目录清理;native stdio 是合成进程,模型调用为零
精确头 quick premerge PASS:直接检查和选中检查零失败;整体仍为 manual_review_required,不能当作合并许可

按当前审查配置没有查询或等待远端 CI。运行时清理可能晚于 IO 期限结束,这是已披露的资源清理边界;不得把它误写成所有工作必须在 25 秒内返回。实账户、模型服务及效果收益未验证,历史失败的具体子阶段也没有被证明。该补丁不能把没有会话或完整恢复点的失败试验自动变成热恢复。

语义与 CI 对齐

复用既有 native stdio 错误与清理 owner;新增分类是该观察命令的局部诊断,不增加 Goal 状态或重规划规则。共享可选参数的默认行为、错误拒绝与原目录不变均有反例覆盖;仓库语义词汇检查通过。公开文档明确披露从单次观察变为同窗有限重试,未偷偷扩大账户、会话或预算权限。

我的整体评价 / Verdict

APPROVE:这是有独立实际入口证据的可靠性增量,恢复了可安全重试的启动通路并改善故障可见性。long_horizon 的改进限于减少尚未开始有效工作的可避免中断,不能外推为分数上涨;user_experience 改善体现在无需额外配置或人工重复尝试,失败仍可辨识。沿现有接收端观察器收敛重试与脱敏,绝对期限由已有共享 stdio reader 执行,避免在 runner 另造通信或恢复状态机。当前切片完整覆盖启动前观察、有限重试、安全诊断及其真实生成命令;正式实验恢复和运行中 Turn 重试保持原 owner。

仅修改 benchmark.runtime.codex_readiness 的 CLI JSON 和共享 stdio 入口,没有网页、布局或其他渲染页面变化。无适用的视觉变更;实际账户和模型效果未验证。生产实现集中在既有 owner,测试主要覆盖边界与默认兼容,没有引入持久重试状态机或新的产品开关。无遗留阻塞发现;后续实验恢复、精确版本采用和合并权限仍应分别验证。

English verdict: APPROVE for the exact head; bounded transport reliability is verified, while live recovery and model-effectiveness claims remain outside this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant